[SPARK-45522][BUILD][CORE][SQL][UI] Migrate from Jetty 9 to Jetty 10 by HiuKwok · Pull Request #43765 · apache/spark

HiuKwok · 2023-11-11T08:39:38Z

What changes were proposed in this pull request?

This is an upgrade ticket to bump the Jetty version from 9 to 10.
This PR aims to bring incremental Jetty upgrades to Spark, as Jetty 9 support already reached EOL.

Why are the changes needed?

Jetty 9 is already beyond EOL, which means that we won't receive any security fix onward for Spark.

Does this PR introduce any user-facing change?

No, SNI host check is now defaulted to true on embedded Jetty, hence set it back to false to maintain backward compatibility.
Despite the redirect behaviour changed for trailing /, but modern browser should be able to pick up the 302 status code and perform redirect accordingly, hence there is no impact on user level.

How was this patch tested?

Junit test case.

Was this patch authored or co-authored using generative AI tooling?

No

HiuKwok · 2024-01-21T13:10:28Z

Checking the build failure

…stead

LuciferYang · 2024-01-25T02:43:25Z

[error] /home/runner/work/spark/spark/core/src/test/scala/org/apache/spark/ui/UISuite.scala:415: File line length exceeds 100 characters

HiuKwok · 2024-01-27T06:18:57Z

I have fixed the code style and other miscellaneous linter errors, right now I'm investigating the Python test case fail, for PySpark connect.

LuciferYang · 2024-01-30T09:57:05Z

cc @srowen @dongjoon-hyun @HyukjinKwon @yaooqinn FYI

srowen · 2024-01-30T13:31:00Z

This is a good step - we should try to get to Jetty 11 next before Spark 4, or even Jetty 12

Co-authored-by: Kent Yao <yao@apache.org>

Co-authored-by: YangJie <yangjie01@baidu.com>

HiuKwok · 2024-01-31T10:11:20Z

Resolved all comments.

srowen · 2024-01-31T15:42:50Z

Merged to master

pan3793 · 2024-05-13T04:10:47Z

@srowen @dongjoon-hyun @LuciferYang @yaooqinn @cloud-fan @HiuKwok

Unfortunately, we identified the compatibility issue on Jetty 11, and it is likely to revert Jetty 11 upgrading, see details in SPARK-48238. If we downgrade Jetty, which version we want to keep, 9 or 10?

I would prefer 9, because I think

both of those versions are End of Community Support, but still receiving critical CVE fixes
some downstream projects may extend Spark UI tabs, e.g. Apache Kyuubi, Apache Gluten, keep using Jetty 9 as Spark 3 does would reduce the potential compatibility issues
Jetty 9 might live longer than Jetty 10

jetty/jetty.project#10103 (comment)

Jetty 9.4 EOL and Jetty 10/11 EOL are only very loosely related.

This is because Java 8 (minimum requirement for Jetty 9.4) currently has longer support from Java vendors than Java 11 (minimum requirement for Jetty 10/11).
Certain vendors support Java 8 until 2030, but Java 11 only until 2026, for example.

For this reason, it may be possible that Jetty 10/11 will go in EOL at a different time than Jetty 9.4, possibly even earlier.

jetty/jetty.project#10485

... specifically that 1 January 2024 will mark the official End of Community Support(EoCS) for Jetty 10 and Jetty 11.

LuciferYang · 2024-05-13T04:17:42Z

@srowen @dongjoon-hyun @LuciferYang @yaooqinn @cloud-fan @HiuKwok

Unfortunately, we identified the compatibility issue on Jetty 11, and it is likely to revert Jetty 11 upgrading, see details in SPARK-48238. If we downgrade Jetty, which version we want to keep, 9 or 10?

I would prefer 9, because I think

both of those versions are End of Community Support, but still receiving critical CVE fixes

some downstream projects may extend Spark UI tabs, e.g. Apache Kyuubi, Apache Gluten, keep using Jetty 9 as Spark 3 does would reduce the potential compatibility issues

Jetty 9 might live longer than Jetty 10

jetty/jetty.project#10103 (comment)

Jetty 9.4 EOL and Jetty 10/11 EOL are only very loosely related.
This is because Java 8 (minimum requirement for Jetty 9.4) currently has longer support from Java vendors than Java 11 (minimum requirement for Jetty 10/11).
Certain vendors support Java 8 until 2030, but Java 11 only until 2026, for example.
For this reason, it may be possible that Jetty 10/11 will go in EOL at a different time than Jetty 9.4, possibly even earlier.

jetty/jetty.project#10485

... specifically that 1 January 2024 will mark the official End of Community Support(EoCS) for Jetty 10 and Jetty 11.

+1 for 9.x

HiuKwok · 2024-05-13T06:11:36Z

@srowen @dongjoon-hyun @LuciferYang @yaooqinn @cloud-fan @HiuKwok

Unfortunately, we identified the compatibility issue on Jetty 11, and it is likely to revert Jetty 11 upgrading, see details in SPARK-48238. If we downgrade Jetty, which version we want to keep, 9 or 10?

I would prefer 9, because I think

both of those versions are End of Community Support, but still receiving critical CVE fixes

some downstream projects may extend Spark UI tabs, e.g. Apache Kyuubi, Apache Gluten, keep using Jetty 9 as Spark 3 does would reduce the potential compatibility issues

Jetty 9 might live longer than Jetty 10

jetty/jetty.project#10103 (comment)

Jetty 9.4 EOL and Jetty 10/11 EOL are only very loosely related.
This is because Java 8 (minimum requirement for Jetty 9.4) currently has longer support from Java vendors than Java 11 (minimum requirement for Jetty 10/11).
Certain vendors support Java 8 until 2030, but Java 11 only until 2026, for example.
For this reason, it may be possible that Jetty 10/11 will go in EOL at a different time than Jetty 9.4, possibly even earlier.

jetty/jetty.project#10485

... specifically that 1 January 2024 will mark the official End of Community Support(EoCS) for Jetty 10 and Jetty 11.

Agree.

cloud-fan · 2024-05-15T07:15:44Z

Thanks for the discussion here! @HiuKwok are you going to do the revert work? If not please let us know and we can help.

### What changes were proposed in this pull request? As a part of [SPARK-55556: Improve Web Security](https://issues.apache.org/jira/browse/SPARK-55556), this PR introduces a new configuration `spark.ui.jetty.sniHostCheckEnabled` that controls Jetty's SNI host check on the Spark UI HTTPS connector. `sniHostCheck` is recommended by Jetty community as we can see that the default value of Jetty is `true` already. The previously Spark-side hardcoded `SecureRequestCustomizer.setSniHostCheck(false)` call in `JettyUtils` is replaced with a value driven by this configuration. The default value is `false`, preserving the existing behavior introduced in SPARK-45522. - #43765 ### Why are the changes needed? In the Jetty usage, `jetty.ssl.sniHostCheck=false` is supposed to override the default behavior. However, since SPARK-45522 (Jetty 10+), Spark has set `SniHostCheck` to `false` strictly to preserve backward compatibility with standalone deployments. Operators who want stricter host checking for security have no way to enable it without patching source. Exposing this as a configuration lets users opt in to SNI host checking when desired. ### Does this PR introduce _any_ user-facing change? No. A new configuration `spark.ui.jetty.sniHostCheckEnabled` (default: `false`) is added in Spark 4.2.0. The default preserves the current behavior, so existing deployments are unaffected. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.7) Closes #55396 from dongjoon-hyun/dongjoon/pensive-hugle-48097c. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

github-actions Bot added WEB UI BUILD CORE labels Nov 11, 2023

LuciferYang reviewed Nov 12, 2023

View reviewed changes

Comment thread pom.xml

LuciferYang reviewed Nov 12, 2023

View reviewed changes

Comment thread core/pom.xml

HiuKwok added 2 commits January 21, 2024 19:09

SPARK-45522: Bump jetty to v10

5ac5460

SPARK-45522: Updat ssl server changes

59809f6

HiuKwok force-pushed the ft-hf-SPARK-45522-jetty-upgradte branch from 0df2668 to 59809f6 Compare January 21, 2024 11:10

POM clean up

3f519ca

HiuKwok changed the title ~~SPARK-45522: [WIP] Jetty 9 -12 upgrade~~ SPARK-45522: Jetty 9 -> 10 upgrade Jan 21, 2024

Remove connector.setStopTimeout as Jetty rely on graceful shutdown in…

1ba384c

…stead

HiuKwok marked this pull request as draft January 23, 2024 11:19

Update ssl factory

786fd60

github-actions Bot added the SQL label Jan 23, 2024

HiuKwok added 2 commits January 24, 2024 23:11

Fix redirect use case

e5b86c1

Revert 34449 test, as the redirect behaviour is now restored on Jetty 10

d0699a0

HiuKwok added 8 commits January 25, 2024 16:46

Test case on proxy

29381f5

SPARK-45522: Update test cases

d179d89

SPARK-45522: Update test

4b7e465

SPARK-45522: Update code naming

749c02d

Update code comment

7443149

Minimise the code diff

73c6127

Update code comment

aa5b1ee

Test-dependecies.sh to comply with new Jetty version format

398c259

Update hadoop deps

1dc6ca8

yaooqinn reviewed Jan 30, 2024

View reviewed changes

Comment thread core/src/main/scala/org/apache/spark/TestUtils.scala Outdated

yaooqinn reviewed Jan 30, 2024

View reviewed changes

Comment thread core/src/main/scala/org/apache/spark/ui/JettyUtils.scala

srowen reviewed Jan 30, 2024

View reviewed changes

Comment thread pom.xml Outdated

LuciferYang reviewed Jan 30, 2024

View reviewed changes

Comment thread core/src/main/scala/org/apache/spark/ui/JettyUtils.scala Outdated

LuciferYang changed the title ~~SPARK-45522: Jetty 9 -> 10 upgrade~~ [SPARK-45522][BUILD][CORE][SQL][UI] Migrate from Jetty 9 to Jetty 10 Jan 30, 2024

HiuKwok and others added 3 commits January 31, 2024 12:45

Spark-45522: Code style

0034d9c

Co-authored-by: Kent Yao <yao@apache.org>

Update code comment

817f517

Co-authored-by: YangJie <yangjie01@baidu.com>

Bump jetty hf version

0b7ecb5

HiuKwok marked this pull request as ready for review January 31, 2024 10:10

HiuKwok requested a review from LuciferYang January 31, 2024 10:11

srowen closed this in 6c19bf6 Jan 31, 2024

dongjoon-hyun mentioned this pull request Apr 17, 2026

[SPARK-56528][CORE] Make Jetty SniHostCheck configurable #55396

Closed

Uh oh!

Conversation

HiuKwok commented Nov 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

Uh oh!

Uh oh!

HiuKwok commented Jan 21, 2024

Uh oh!

LuciferYang commented Jan 25, 2024

Uh oh!

HiuKwok commented Jan 27, 2024

Uh oh!

LuciferYang commented Jan 30, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

srowen commented Jan 30, 2024

Uh oh!

HiuKwok commented Jan 31, 2024

Uh oh!

srowen commented Jan 31, 2024

Uh oh!

pan3793 commented May 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LuciferYang commented May 13, 2024

Uh oh!

HiuKwok commented May 13, 2024

Uh oh!

cloud-fan commented May 15, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

HiuKwok commented Nov 11, 2023 •

edited

Loading

pan3793 commented May 13, 2024 •

edited

Loading